Machine learning system and method, integration server, information processing apparatus, program, and inference model creation method

ABSTRACT

Provided are a machine learning system and method, an integration server, an information processing apparatus, a program, and an inference model creation method capable of solving a problem caused by non-uniform quality of learning data in federated learning and improving an inference accuracy of a model. Each of a plurality of client terminals classifies data stored in a medical institution based on a data acquisition condition and classifies learning data into each data group acquired under the same or a similar acquisition condition. Each client terminal executes machine learning of a learning model for each learning data group classified into each condition category and transmits each learning result and condition information to an integration server. The integration server integrates received learning results for each condition category to create a plurality of master model candidates and evaluates the inference accuracy of each master model candidate.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation of PCT InternationalApplication No. PCT/JP2020/038694 filed on Oct. 14, 2020 claimingpriority under 35 U.S.C § 119(a) to Japanese Patent Application No.2019-192548 filed on Oct. 23, 2019. Each of the above applications ishereby expressly incorporated by reference, in its entirety, into thepresent application.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a machine learning system and method,an integration server, an information processing apparatus, a program,and an inference model creation method, and particularly relates to amachine learning technique using a federated learning mechanism.

2. Description of the Related Art

In development of medical artificial intelligence (AI) using deeplearning, it is necessary to train an AI model. However, for thislearning, it is necessary to extract learning data such as a diagnosisimage from a medical institution to an external development site or toan external development server. For this reason, there are few medicalinstitutions that can cooperate in providing learning data. Further,even in a case where learning data is provided from a medicalinstitution, there is always a privacy-related risk.

On the other hand, in a case where a federated learning mechanism isused, a federated learning mechanism being proposed in H. BrendanMcMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera yArcas, “Communication-Efficient Learning of Deep Networks fromDecentralized Data”, arXiv:1602.05629v3 [cs.LG], 28 Feb. 2017, learningis performed on a terminal in which data for training exists, and only aweight parameter of a network model that is a learning result on eachterminal is transmitted from a terminal group to an integration server.That is, in federated learning, learning data is not provided to theintegration server side, and only data of the learning result on eachterminal is provided from the terminal side to the integration serverside.

For this reason, learning can be performed without extracting data thatrequires consideration for privacy to the outside. Thus, federatedlearning is a technique that has been attracting attention in recentyears.

In Micah J Sheller, G Anthony Reina, Brandon Edwards, Jason Martin, andSpyridon Bakas, “Multi-Institutional Deep Learning Modeling WithoutSharing Patient Data: A Feasibility Study on Brain Tumor Segmentation”,arXiv:1810.04304v2 [cs.LG], 22 Oct. 2018, a result of an example inwhich federated learning is applied to development of a medical AI isreported.

SUMMARY OF THE INVENTION

In a case where federated learning is used for development of a medicalAI, it is not necessary to extract data such as a diagnosis image.However, contents of the data held by each medical institution vary, andlearning environments are different for each client. As a result,results of learning performed by each client also vary. For example, ina case where an AI model for medical image diagnosis is created by anunspecified large number of medical institutions participating inlearning, issues are that imaging conditions are not constant for eachimage, setting criteria for the imaging conditions are different foreach medical institution, and thus there is no specific nature of noiseon the image. This fact makes it difficult to separate, in a case wherean inference accuracy of the AI model is not improved that much bylearning in a case where the AI model for medical image diagnosis iscreated, whether a cause of the problem is noise or the model itself.The nature of noise changes depending on various conditions such assettings at the time of imaging. Therefore, it is difficult to develop ageneral-purpose and effective noise removal method.

In the existing federated learning mechanism, there is no indexregarding client selection, such as how to select clients used for thelearning from a large number of clients. In a case where the clientsused for the learning are randomly selected from the client populationin the federated learning mechanism, the above problem caused by imageswith different imaging conditions still occurs.

That is, due to the cause such as the difficult noise removal by variouschanges in the imaging conditions for each inspection image ornon-uniform quality of learning images, it may be difficult to improvethe inference accuracy of the model in a case where the training of theAI model is performed.

Such a problem is not limited to the AI model for medical images, but isalso a common problem for an AI model that uses data other than imagessuch as electrocardiogram waveform data. The term “imaging condition”can be extended to a term such as “inspection condition” or “dataacquisition condition”.

The present invention has been made in view of such circumstances, andan object of the present invention is to provide a technique of solvinga problem caused by non-uniform quality of learning data in a case wherea federated learning mechanism for performing training of an AI modelcan be implemented without extracting personal information such as adiagnosis image that requires consideration for privacy from a medicalinstitution to the outside, and to provide a machine learning system andmethod, an integration server, an information processing apparatus, aprogram, and an inference model creation method capable of improving aninference accuracy of a model.

A machine learning system according to one aspect of the presentdisclosure includes a plurality of client terminals and an integrationserver. Each of the plurality of client terminals includes a learningdata classification processing unit that classifies data stored in adata storage apparatus of a medical institution based on an acquisitioncondition of the data to classify learning data into each data groupacquired under the same or a similar acquisition condition, a learningprocessing unit that executes machine learning of a learning model foreach learning data group classified by the learning data classificationprocessing unit into each condition category of the same or a similaracquisition condition, and a transmission unit that transmits learningresults of the learning model executed for each learning data group andcondition information regarding the acquisition condition of thelearning data group used for the learning, to the integration server.The integration server includes a trained master model, asynchronization processing unit that synchronizes the learning model ofeach client terminal side with the master model before each of theplurality of client terminals trains the learning model, a receptionunit that receives the learning results of the learning model and thecondition information from each of the plurality of client terminals, aclassification processing unit that classifies the learning results intoeach condition category, a master model candidate creation unit thatintegrates the learning results for each condition category to create aplurality of master model candidates, and an accuracy evaluation unitthat evaluates an inference accuracy of each of the plurality of mastermodel candidates.

According to the present aspect, the learning data group classified intoeach condition category based on the data acquisition condition is acollection of the data obtained under substantially uniform conditionsin which data acquisition conditions fall into a category of the same ora similar condition. Each client terminal performs the training of eachlearning model for each condition category using the learning data groupclassified into each condition category and transmits each learningresult and condition information to the integration server. Theintegration server classifies the received learning results into eachcondition category and integrates the learning results for eachcondition category to create the plurality of master model candidates.With the evaluation of the inference accuracy of each of the pluralityof master model candidates created in this way, it is possible toextract a model with good inference accuracy.

According to the present aspect, the data of the learning data groupused for the learning is homogenized, and thus effective learning can beperformed.

The term “plurality of client terminals” may be an unspecified largenumber of client terminals. The client terminal may be configured toinclude a “data storage apparatus of a medical institution”, or the“data storage apparatus of a medical institution” and the “clientterminal” may be separate apparatuses.

For the term “data acquisition condition”, the “acquisition condition”may be, for example, a type of apparatus used to generate the data andsome or all of the various parameters set in the apparatus may beparaphrased as a “generation condition”. The term “condition category”includes the concept of terms such as a condition division, a conditionclassification label, a condition type, a condition frame, and acondition range. The condition category may be set with including arange of a similar acquisition condition or may be set without includingthe range of a similar acquisition condition. The description “the sameor a similar acquisition condition” may include only the sameacquisition condition or may include the same and similar acquisitioncondition.

In the machine learning system according to another aspect of thepresent disclosure, the data may include an image captured by using animaging apparatus, and the acquisition condition may include an imagingcondition for the image.

In the machine learning system according to still another aspect of thepresent disclosure, the imaging condition may include a conditionregarding a model of the imaging apparatus used for imaging.

The condition regarding the model of the imaging apparatus may bespecified by, for example, a model name and/or a model number of theimaging apparatus.

In the machine learning system according to still another aspect of thepresent disclosure, the imaging condition may include a condition of animaging parameter settable at a time of imaging.

The imaging parameter includes, for example, a radiation dose. Theimaging condition may be a combination of a plurality of conditions.

In the machine learning system according to still another aspect of thepresent disclosure, the data may include inspection data acquired byusing an inspection apparatus, and the acquisition condition may includean inspection condition under which the inspection data is acquired.

The imaging apparatus is a form of the inspection apparatus.

In the machine learning system according to still another aspect of thepresent disclosure, the acquisition condition may include a conditionregarding a value of a parameter settable in an apparatus used to obtainthe data, and the learning data classification processing unit mayclassify acquisition conditions in which a specific value of theparameter, which is a specific acquisition condition, is within adesignated value range into the condition category in which theacquisition conditions are handled as the acquisition condition that isthe same as or similar to the specific acquisition condition.

Each of the imaging apparatus and the inspection apparatus is a form of“apparatus used to obtain data”. The range of the condition belonging tothe condition category handled as the same or a similar condition may bedesignated in advance or may be dynamically designated according to aninstruction from the integration server or the like.

In the machine learning system according to still another aspect of thepresent disclosure, a combination of conditions in which a plurality ofacquisition conditions are handled as a similar condition may bedesignated, and the learning data classification processing unit mayperform the classification into the condition category according to asetting of the designated similar condition.

In the machine learning system according to still another aspect of thepresent disclosure, each of the plurality of client terminals may be aterminal provided in a medical institution network of a differentmedical institution.

In the machine learning system according to still another aspect of thepresent disclosure, the integration server may be provided in a medicalinstitution network or outside the medical institution network.

In the machine learning system according to still another aspect of thepresent disclosure, the learning results transmitted from the clientterminal to the integration server may include a weight parameter of thelearning model after the learning.

In the machine learning system according to still another aspect of thepresent disclosure, the data used as the learning data may include atleast one type of data among a two-dimensional image, athree-dimensional image, a moving image, time-series data, or documentdata.

In the machine learning system according to still another aspect of thepresent disclosure, each model of the learning model, the master model,and the master model candidate may be configured by using a neuralnetwork.

An appropriate network model is applied according to a type of thelearning data and a type of data that is input in the inference.

In the machine learning system according to still another aspect of thepresent disclosure, the data used as the learning data may include atwo-dimensional image, a three-dimensional image, or a moving image, andeach model of the learning model, the master model, and the master modelcandidate may be configured by using a convolutional neural network.

In the machine learning system according to still another aspect of thepresent disclosure, the data used as the learning data may includetime-series data or document data, and each model of the learning model,the master model, and the master model candidate may be configured byusing a recurrent neural network.

In the machine learning system according to still another aspect of thepresent disclosure, the integration server may further include aninformation storage unit that stores information indicating acorrespondence relationship as to which client cluster among a pluralityof client clusters each of the plurality of master model candidatescreated is based on.

In the machine learning system according to still another aspect of thepresent disclosure, the integration server may further include a displaydevice on which information indicating a progress status of learning ofeach of the master model candidates is displayed.

In the machine learning system according to still another aspect of thepresent disclosure, a verification data storage unit that storesverification data classified based on a data acquisition condition isfurther included, and the accuracy evaluation unit may evaluate theinference accuracy of the master model candidate using the verificationdata.

The verification data storage unit may be included in the integrationserver or may be an external storage apparatus connected to theintegration server.

In the machine learning system according to still another aspect of thepresent disclosure, the accuracy evaluation unit may include aninference accuracy calculation unit that compares an inference resultoutput from the master model candidate by inputting verification data tothe master model candidate with correct answer data of the verificationdata to calculate the inference accuracy of the master model candidate,and an accuracy target value comparison unit that compares the inferenceaccuracy of the master model candidate with an accuracy target value.

A machine learning method according to still another aspect of thepresent disclosure uses a plurality of client terminals and anintegration server. The machine learning method includes classifying,via each of the plurality of client terminals, data stored in a datastorage apparatus of each of different medical institutions based on anacquisition condition of the data to classify learning data into eachdata group acquired under the same or a similar acquisition condition,synchronizing a learning model of each client terminal side with atrained master model stored in the integration server before each of theplurality of client terminals trains the learning model, executing, viaeach of the plurality of client terminals, machine learning of thelearning model for each learning data group classified into eachcondition category of the same or a similar acquisition condition,transmitting, via each of the plurality of client terminals, learningresults of the learning model executed for each learning data group andcondition information regarding the acquisition condition of thelearning data group used for the learning, to the integration server,and, via the integration server, receiving the learning results of thelearning model and the condition information from each of the pluralityof client terminals, classifying the learning results into eachcondition category, integrating the learning results for each conditioncategory to create a plurality of master model candidates, andevaluating an inference accuracy of each of the plurality of mastermodel candidates.

An integration server according to another aspect of the presentdisclosure is connected to a plurality of client terminals via acommunication line. The integration server includes a master modelstorage unit that stores a trained master model, a synchronizationprocessing unit that synchronizes a learning model of each clientterminal with the master model before each of the plurality of clientterminals trains the learning model, a reception unit that receiveslearning results of the learning model and condition informationregarding an acquisition condition of data included in a learning datagroup used for the learning from each of the plurality of clientterminals, a classification processing unit that classifies the learningresults into each condition category in which the acquisition conditionis handled as the same or a similar condition, a master model candidatecreation unit that integrates the learning results for each conditioncategory to create a plurality of master model candidates, and anaccuracy evaluation unit that evaluates an inference accuracy of each ofthe plurality of master model candidates.

The integration server according to another aspect of the presentdisclosure is connected to a plurality of client terminals via acommunication line. The integration server includes a first processorand a first computer-readable medium, which is a non-transitory tangiblemedium, on which a first program executed by the first processor isrecorded. The first processor executes, according to an instruction ofthe first program, processing including storing a trained master modelon the first computer-readable medium, synchronizing a learning model ofeach client terminal side with the master model before each of theplurality of client terminals trains the learning model, receivinglearning results of the learning model and condition informationregarding an acquisition condition of data included in a learning datagroup used for the learning from each of the plurality of clientterminals, classifying the learning results into each condition categoryin which the acquisition condition is handled as the same or a similarcondition, integrating the learning results for each condition categoryto create a plurality of master model candidates, and evaluating aninference accuracy of each of the plurality of master model candidates.

A program according to still another aspect of the present disclosure isa program for causing a computer to function as an integration serverconnected to a plurality of client terminals via a communication line.The program causes the computer to realize a function of storing atrained master model, a function of synchronizing a learning model ofeach client terminal side with the master model before each of theplurality of client terminals trains the learning model, a function ofreceiving learning results of the learning model and conditioninformation regarding an acquisition condition of data included in alearning data group used for the learning from each of the plurality ofclient terminals, a function of classifying the learning results intoeach condition category in which the acquisition condition is handled asthe same or a similar condition, a function of integrating the learningresults for each condition category to create a plurality of mastermodel candidates, and a function of evaluating an inference accuracy ofeach of the plurality of master model candidates.

An information processing apparatus according to another aspect of thepresent disclosure is used as a client terminal connected to anintegration server via a communication line. The information processingapparatus includes a learning data classification processing unit thatclassifies data stored in a data storage apparatus of a medicalinstitution based on an acquisition condition of the data to classifylearning data into each data group acquired under the same or a similaracquisition condition, a learning processing unit that executes, with alearning model synchronized with a master model stored in theintegration server as the learning model in an initial state beforelearning starts, machine learning of the learning model for eachlearning data group classified by the learning data classificationprocessing unit into each condition category of the same or a similaracquisition condition, and a transmission unit that transmits learningresults of the learning model executed for each learning data group andcondition information regarding the acquisition condition of thelearning data group used for the learning, to the integration server.

An information processing apparatus according to another aspect of thepresent disclosure is used as a client terminal connected to anintegration server via a communication line. The information processingapparatus includes a second processor and a second computer-readablemedium, which is a non-transitory tangible medium, on which a secondprogram executed by the second processor is recorded. The secondprocessor executes, according to an instruction of the second program,processing including classifying data stored in a data storage apparatusof a medical institution based on an acquisition condition of the datato classify learning data into each data group acquired under the sameor a similar acquisition condition, executing, with a learning modelsynchronized with a master model stored in the integration server as thelearning model in an initial state before learning starts, machinelearning of the learning model for each learning data group classifiedinto each condition category of the same or a similar acquisitioncondition, and transmitting learning results of the learning modelexecuted for each learning data group and condition informationregarding the acquisition condition of the learning data group used forthe learning, to the integration server.

A program according to another aspect of the present disclosure is aprogram for causing a computer to function as a client terminalconnected to an integration server via a communication line. The programcauses the computer to realize a function of classifying data stored ina data storage apparatus of a medical institution based on anacquisition condition of the data to classify learning data into eachdata group acquired under the same or a similar acquisition condition, afunction of executing, with a learning model synchronized with a mastermodel stored in the integration server as the learning model in aninitial state before learning starts, machine learning of the learningmodel for each learning data group classified into each conditioncategory of the same or a similar acquisition condition, and a functionof transmitting learning results of the learning model executed for eachlearning data group and condition information regarding the acquisitioncondition of the learning data group used for the learning, to theintegration server.

A method of creating an inference model according to still anotheraspect of the present disclosure is an inference model creation methodby performing machine learning using a plurality of client terminals andan integration server. The inference model creation method includesclassifying, via each of the plurality of client terminals, data storedin a data storage apparatus of each of different medical institutionsbased on an acquisition condition of the data to classify learning datainto each data group acquired under the same or a similar acquisitioncondition, synchronizing a learning model of each client terminal sidewith a trained master model stored in the integration server before eachof the plurality of client terminals trains the learning model,executing, via each of the plurality of client terminals, machinelearning of the learning model for each learning data group classifiedinto each condition category of the same or a similar acquisitioncondition, transmitting, via each of the plurality of client terminals,learning results of the learning model executed for each learning datagroup and condition information regarding the acquisition condition ofthe learning data group used for the learning, to the integrationserver, and, via the integration server, receiving the learning resultsof the learning model and the condition information from each of theplurality of client terminals, classifying the learning results intoeach condition category, integrating the learning results for eachcondition category to create a plurality of master model candidates,evaluating an inference accuracy of each of the plurality of mastermodel candidates, and creating an inference model with higher inferenceaccuracy than the master model based on a model whose inference accuracysatisfies a target accuracy among the plurality of master modelcandidates.

The inference model creation method is understood as an invention of amethod of producing the inference model. The term “inference” includesconcepts of prediction, estimation, classification, and determination.The inference model may be paraphrased as an “AI model”.

According to the present invention, it is possible to perform thelearning for each learning data group in which the quality of the dataused for the learning is substantially homogenized. Accordingly, it ispossible to efficiently perform the learning and thus to improve theinference accuracy of the model.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a conceptual diagram illustrating an outline of a machinelearning system according to an embodiment of the present invention.

FIG. 2 is a conceptual diagram illustrating an example of local learningperformed for each client.

FIG. 3 is a conceptual diagram illustrating an example of creationprocessing of a master model candidate performed on an integrationserver.

FIG. 4 is a diagram schematically illustrating a system configurationexample of the machine learning system according to the embodiment ofthe present invention.

FIG. 5 is a block diagram illustrating a configuration example of anintegration server.

FIG. 6 is a block diagram illustrating a configuration example of acomputer aided detection/diagnosis (CAD) server as an example of aclient.

FIG. 7 is a flowchart illustrating an example of an operation of aclient terminal based on a local learning management program.

FIG. 8 is a flowchart illustrating an example of an operation of theintegration server based on a master model learning management program.

FIG. 9 is a flowchart illustrating an example of processing ofevaluating an inference accuracy of a master model candidate in theintegration server.

FIG. 10 is a block diagram illustrating an example of a hardwareconfiguration of a computer.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, preferred embodiments of the present invention will bedescribed with reference to the accompanying drawings.

Outline of Machine Learning System

FIG. 1 is a conceptual diagram illustrating an outline of a machinelearning system according to an embodiment of the present invention. Amachine learning system 10 is a computer system that performs machinelearning using a federated learning mechanism. The machine learningsystem 10 includes a plurality of clients 20 and an integration server30. Federated learning is sometimes referred to as “federationlearning”, “cooperative learning”, or “combination learning”.

Each of the plurality of clients 20 illustrated in FIG. 1 indicates amedical institution terminal that is provided on a network in a medicalinstitution such as a hospital. Here, the term “terminal” refers to acomputing resource existing in a network that can safely access data ina medical institution, and the terminal may not physically exist in themedical institution. That is, the client 20 may be a physical machine ora virtual machine, and the specific form thereof is not limited. Theclient 20 is an example of a “client terminal” according to the presentdisclosure. A computer network in a medical institution is referred toas a “medical institution network”.

It is assumed that each client 20 exists for each data group fortraining an AI model. The term “for each data group” described hereinmay be understood as “for each medical institution” that holds a datagroup to be used for the training of the AI model. That is, it isassumed that one client exists for one medical institution.

In order to distinguish and display each of the plurality of clients 20,representations such as “Client 1” and “Client 2” are used in FIG. 1 andsubsequent drawings. A number after “Client” is an index as anidentification number for identifying each client 20. In the presentspecification, the client 20 having an index of m is represented by a“client CLm”. For example, the client CL1 represents “Client 1” inFIG. 1. m corresponds to a client identification number (ID number).Assuming that a total number of the clients 20 managed by theintegration server 30 is M, m represents an integer equal to or largerthan 1 and equal to or smaller than M. In FIG. 1, the clients 20 havingindexes from m=1 to m=N+1 are illustrated. N represents an integer equalto or larger than 2. An entire set of the clients 20 having a totalnumber of M and participating in the learning is referred to as a“learning client group” or a “client population”.

Each client 20 holds local data LD in a storage apparatus of a localclient. The local data LD is a data group accumulated by a medicalinstitution to which the client 20 belongs.

Each client 20 includes a local learning management program 21, which isa distribution learning client program. Each client 20 performs aniteration for the training of a local model LM using the local data LDof the local client according to the local learning management program21.

The local learning management program 21 is a program installed in eachclient 20. The local learning management program 21 classifies imagegroups for each similar imaging condition in the client 20 or managessynchronization between the local model LM and a master model on theintegration server 30.

Here, the term “imaging condition” may be all or some of imagingparameters that can be set at the time of imaging, such as a type ofapparatus in use, a model number of an imaging apparatus used forimaging and a radiation dose.

The term “for each similar imaging condition” means for each imagingcondition group in a case where one or a plurality of conditions thatfall into a category of similar imaging conditions are collectivelydefined as the same imaging condition group. The term “group” can berephrased as a term such as “category”, “type”, “class”, “group”, or“classification division”. The category of similar imaging conditionscan be designated as appropriate according to a type of data to betargeted and/or a task of inference.

In the present specification, a unit of classification of the imagingcondition group, which is collected as similar imaging conditions, isreferred to as an “imaging condition category”. The description “foreach similar imaging condition” is synonymous with the description “foreach imaging condition category” or “for each identical imagingcondition”. The imaging condition category may be defined as a specificimaging condition that does not include a similar range. The imagingcondition category is an example of the “condition category” in thepresent disclosure.

The local learning management program 21 classifies the local data LDfor each imaging condition category. Further, the local learningmanagement program 21 creates the local model LM for each data group ofthe classified local data LD and performs the training of thecorresponding local model LM by using the classified local data LD.

The local model LM is, for example, an AI model for medical imagediagnosis that is incorporated in a CAD system. The term “CAD” includesconcepts of both computer aided detection (CADe) and computer aideddiagnosis (CADx). The local model LM is configured using, for example, ahierarchical multi-layer neural network. In the local model LM, anetwork weight parameter is updated by deep learning using the localdata LD as learning data. The weight parameter includes a filtercoefficient (weight of a connection between nodes) of a filter used forprocessing of each layer and a bias of a node. The local model LM is anexample of a “learning model of client terminal side” according to thepresent disclosure.

The term “neural network” is a mathematical model for informationprocessing that simulates a mechanism of a brain-nervous system.Processing using the neural network can be realized by using a computer.A processing unit including the neural network may be configured as aprogram module.

As a network structure of the neural network used for the learning, anappropriate network structure is employed according to a type of dataused for input. The AI model for medical image diagnosis may beconfigured using, for example, various convolutional neural networks(CNNs) having a convolutional layer. The AI model that handlestime-series data, document data, or the like may be configured using,for example, various recurrent neural networks (RNNs).

The plurality of clients 20 are connected to the integration server 30via a communication network. The client 20 transmits a learning resultof each local model LM and imaging condition information of the dataused for the learning to the integration server 30. The imagingcondition information may be information representing the imagingcondition category, information indicating actual imaging conditions, ora combination of the pieces of information. The imaging conditioninformation is an example of “condition information” in the presentdisclosure.

The integration server 30 receives each learning result and the imagingcondition information from the plurality of clients 20, classifies thelearning results for each imaging condition category, and integrates thelearning results for each imaging condition category to create aplurality of master model candidates MMC. Further, the integrationserver 30 performs processing of evaluating an inference accuracy ofeach created master model candidate MMC.

A location of the integration server 30 may be on a computer network onwhich an entity developing the AI model has access rights, and a form ofthe server may be a physical server, a virtual server, or the like. Theintegration server 30 may be provided in a medical institution network,or may be provided outside a medical institution network. For example,the integration server 30 may be provided in a company that is locatedgeographically away from a medical institution and that develops medicalAI or may be provided on a cloud.

The integration server 30 includes a master model learning managementprogram 33. The master model learning management program 33 classifiesand collects the learning results transmitted from the client 20 foreach similar imaging condition (imaging condition category) and createsthe master model candidate MMC for each imaging condition category.Collectively grouping the learning results for each imaging conditioncategory is equivalent to grouping the clients 20, which are thetransmission sources of the learning results, for each imaging conditioncategory to create a client cluster. The client cluster is a part of theclient group extracted from the client population. In the presentspecification, the client group which is a combination of the clients 20used to create the master model candidate MMC is referred to as “clientcluster”.

Here, K types of imaging condition categories are assumed to be set. Kis an integer equal to or larger than 2. The integration server 30creates K groups of the client clusters from the client population andcreates K master model candidates MMC by integrating the learningresults for each client cluster. One client 20 may be an element of aplurality of client clusters. The number of clients 20 (the number ofclients) constituting each of the K groups of the client clusters ispreferably the same, but may be different.

In FIG. 1, the clients CL1, CL2, and CL3 belong to the same clientcluster, and the clients CL4, CLN, and CLN+1 belong to the same clientcluster.

In FIG. 1, an arrow extending from a left side of a circle surrounding adisplay “Federated Avg” indicates that data of the trained local modelLM is transmitted from each client 20 belonging to the same clientcluster. The data of the local model LM as a learning result providedfrom each client 20 to the integration server 30 may be the weightparameter of the trained local model LM.

The circle surrounding the display “Federated Avg” represents processingof integrating the learning results. In the processing, the weightstransmitted from each client 20 are integrated by averaging or the liketo create the master model candidate MMC, which is an integration model.A method of integration processing is not limited to a simple arithmeticmean. The integration may be performed with the weights based on factorssuch as an attribute of the client 20, a past integration result, thenumber of pieces of data for each medical institution used forre-learning, and a level of a medical institution evaluated by people.

In FIG. 1, “Master model 1” and “Master model K” illustrated at ends ofarrows extending to a right side of the circle surrounding the display“Federated Avg” indicate the master model candidates MMC created fromeach client cluster. Assuming that an index for identifying each of theK groups of the client clusters is k, k represents an integer equal toor larger than 1 and equal to or smaller than K. The index “k” may beunderstood as an index that identifies the K types of imaging conditioncategories. In the present specification, the master model candidate MMCcreated by integrating the learning results of each client 20 belongingto the client cluster having the index k may be represented as “MMCk”.For example, the master model candidate MMC1 represents “Master model 1”in FIG. 1.

The master model learning management program 33 of the integrationserver 30 evaluates the inference accuracy of the master model candidateMMC using verification data for each of various imaging conditioncategories prepared in advance. The verification data for each of thevarious imaging condition categories may be stored in an internalstorage apparatus of the integration server 30 or may be stored in anexternal storage apparatus connected to the integration server 30.

The integration server 30 repeats the learning in the client 20 and thetraining of the master model candidate MMC for each identical imagingcondition until each master model candidate MMC achieves a desiredinference accuracy.

FIG. 2 is a conceptual diagram illustrating an example of local learningperformed for each client 20. FIG. 2 shows an example of the locallearning performed in the clients CL1, CL2, and CL3. In FIG. 2, therepresentation of local data LDm represents local data held by themedical institution to which the client CLm specified by the index mbelongs. The representation of local data LDmk is data collected byclassifying the local data LDm for each similar imaging condition andrepresents the local data of the imaging condition category specified bythe index k. The representation of a local model LMmk represents a localmodel on which the learning using the local data LDmk is performed inthe client CLm.

The local data LD1 of the client CL1 shown in FIG. 2 is classified intoeach similar imaging condition. For example, the local data LD1 of theclient CL1 is divided into local data LD11 including an image groupimaged under imaging conditions belonging to a first imaging conditioncategory and local data LD12 including an image group imaged underimaging conditions belonging to a second imaging condition category.

The client CL1 performs the training of a local model LM11 using thelocal data LD11 as the learning data and performs the training of alocal model LM12 using the local data LD12 as the learning data.

The local data LD2 of the client CL2 shown in FIG. 2 includes, forexample, local data LD21 including an image group imaged under imagingconditions belonging to the first imaging condition category, local dataLD23 including an image group imaged under imaging conditions belongingto a third imaging condition category, and local data LD24 including animage group imaged under imaging conditions belonging to a fourthimaging condition category. The client CL2 performs the training of alocal model LM21 using the local data LD21 as the learning data,performs the training of a local model LM23 using the local data LD23 asthe learning data, and performs the training of a local model LM24 usingthe local data LD24 as the learning data.

The local data LD3 of the client CL3 shown in FIG. 2 is divided into,for example, local data LD32 including an image group imaged underimaging conditions belonging to the second imaging condition categoryand local data LD33 including an image group imaged under imagingconditions belonging to the third imaging condition category. The clientCL3 performs the training of a local model LM32 using the local dataLD32 as the learning data and performs the training of a local modelLM33 using the local data LD33 as the learning data.

The number of local models LMmk to be trained in each client CLm is notlimited to the example of FIG. 2 and may have various forms depending oncontents of the local data LDm. In a case where the local data LDm ofthe client CLm includes the local data LDmk of a k-th imaging conditioncategory, the client CLm can perform the training of the local modelLMmk using the local data LDmk as the learning data. Each client CLm canperform the training of one or more local models LMmk. The local dataLDmk is an example of the “data group acquired under the same or asimilar acquisition condition” and the “learning data group classifiedinto each condition category” in the present disclosure.

FIG. 3 is a conceptual diagram illustrating an example of creationprocessing of the master model candidate MMC performed on theintegration server 30. FIG. 3 illustrates a state where the learningresults and the imaging condition information are transmitted from theclients CL1, CL2, and CL3 illustrated in FIG. 2 to the integrationserver 30.

The integration server 30 includes an imaging condition classificationunit 332 and a master model candidate creation unit 334. The imagingcondition classification unit 332 classifies the learning result datatransmitted from the client CLm for each imaging condition category tocreate the client cluster for each imaging condition category. A tabledisplayed in a frame of the imaging condition classification unit 332 inFIG. 3 conceptually illustrates a collection of classified and organizeddata. Numbers “1”, “2”, and “3” in a column displayed as “imagingcondition” in the same table indicate values of index “k” of the imagingcondition category. A collection of the clients CLm displayed in cellsof a column displayed as “Client” corresponds to the client cluster. Theimaging condition classification unit 332 is an example of a“classification processing unit” in the present disclosure.

The master model candidate creation unit 334 integrates the learningresults of the client clusters for each imaging condition categoryclassified by the imaging condition classification unit 332 to create amaster model candidate MMCk. That is, the master model candidatecreation unit 334 creates the master model candidate MMCk using thelearning results classified into the k-th imaging condition category.

Outline of Machine Learning Method

An example of a machine learning method by the machine learning system10 according to the embodiment of the present invention will bedescribed. The machine learning system 10 operates according toprocedure 1 to procedure 11 described below.

[Procedure 1] As illustrated in FIG. 1, the distribution learning clientprogram for federated learning is executed on the medical institutionterminal (client 20) that is provided on the computer network of themedical institution in which the data group for training the AI modelexists.

[Procedure 2] In each medical institution network, the image used forthe learning is stored together with the imaging condition in a placeaccessible from the local learning management program 21.

[Procedure 3] The local learning management program 21 of the client 20determines whether or not the local model LM exists on the terminal ofthe client 20. In a case where the local model LM does not exist, therequired number of learning images is collected and classified into eachsimilar imaging condition from the local data LD. In a similarclassification handled as “similar imaging condition” in this case,imaging conditions in which a specific imaging condition (for example,specific radiation dose) according to the purpose of learning is withina designated value range may be classified into the same conditiondivision (imaging condition category) as “similar imaging condition”, orconditions in which a plurality of different imaging conditions arehandled as a similar condition may be set, and imaging conditionsaccording to the setting may be classified into the same conditiondivision as “similar imaging condition”. The radiation dose is anexample of “imaging parameter that can be set at the time of imaging”and “parameter that can be set in the apparatus used to obtain data” inthe present disclosure. A value of the radiation dose is an example of“imaging parameter value” and “parameter value” in the presentdisclosure.

On the other hand, a case where the local model LM already exists meansthat the image classification for each similar imaging condition hasalready been performed. Therefore, there may be no need to newly createthe learning data group for each imaging condition category for thelearning.

[Procedure 4] The integration server 30 synchronizes, in a case wherethe latest version of the master model for the learning exists, thelatest version of the master model with the local model LM on eachclient 20 before each of the plurality of clients 20 starts thelearning. The master model is a trained AI model. In this case, arelationship between the local model LM and the master model, which aresynchronized, is maintained in the subsequent learning process.

[Procedure 5] In the client 20, the local model LM is created for eachimaging condition category created in procedure 3 according to the locallearning management program 21, and the training of each local model LMproceeds for each imaging condition category. In this case, the trainingof each local model LM may proceed by the designated number ofiterations or may proceed until designated accuracy improvement issatisfied with the designated upper limit number of iterations as anupper limit. Here, the learning “for each imaging condition category”means that conditions of the data (learning data) used for the learningare closer to uniform than those of random extraction, and thus it isexpected that the learning is effectively performed by thehomogenization of the learning data.

[Procedure 6] The client 20 transmits, in a case where the training ofthe local model LM ends, the learning result of the local model LM tothe integration server 30 in association with the imaging conditions ofthe data group used for the learning. The learning result transmittedfrom the client 20 to the integration server 30 may be the weightparameter of the trained local model LM. The data of the weightparameter after learning that is transmitted from the client 20 to theintegration server 30 may be a difference from the weight parameter ofthe latest version of the master model synchronized with the integrationserver 30.

[Procedure 7] The integration server 30 stores the learning resulttransmitted from each client 20 and metadata such as the imagingcondition accompanying the learning result in a data storage unit suchas a database.

[Procedure 8] The master model learning management program 33 on theintegration server 30 creates the classification of similar imagingconditions based on the learning result of the client 20 and themetadata accompanying the learning result which are stored in procedure7. Further, the master model learning management program 33 creates theclient cluster for each of the classified similar imaging conditions(for each imaging condition category) and creates the master modelcandidate MMC from each of the client clusters. Accordingly, theplurality of master model candidate MMCs are created. The master modellearning management program 33 stores a correspondence relationship asto from which client cluster each master model candidate MMC is createdin the data storage unit such as the database. The relationship(correspondence relationship) between the client cluster and the mastermodel candidate MMC is maintained until the subsequent training of themaster model candidate MMC ends.

[Procedure 9] The master model learning management program 33 evaluatesthe inference accuracy for each master model candidate MMC, using theverification data for each imaging condition prepared in advance on theintegration server 30 or in an environment where the integration server30 has access. That is, the master model learning management program 33causes the master model candidate MMC to perform an inference by usingthe verification data as an input to the master model candidate MMC,compares an inference result with correct answer data to calculate theinference accuracy, and stores the inference accuracy of the mastermodel candidate MMC in the data storage unit such as the database.

[Procedure 10] In a case where the inference accuracy of the mastermodel candidate MMC exceeds a target accuracy as a result of measuringthe accuracy of the master model candidate MMC in procedure 9, themaster model learning management program 33 ends the training of themaster model candidate MMC and sends notification of the end of thelearning thereof.

[Procedure 11] In a case where the inference accuracy of the mastermodel candidate MMC does not reach the target accuracy as a result ofmeasuring the accuracy of the master model candidate MMC in procedure 9,the local model LM of the client cluster used to create the master modelcandidate MMC is synchronized with the master model candidate MMC usingthe information stored in procedure 8, and the procedures 5 to 11 arerepeated. In the synchronization processing of procedure 11, the mastermodel candidate MMC having the highest inference accuracy among theplurality of master model candidate MMCs may be synchronized with thelocal model LM.

Accordingly, it is possible to create the master model candidate MMC byintegrating the learning results with the image group acquired undersubstantially uniform imaging conditions for each imaging conditioncategory as the learning data. According to the present embodiment, withthe homogenization of the imaging conditions of the learning data, it ispossible to efficiently perform the learning and to improve theinference accuracy of the model. With the implementation of the machinelearning method using the machine learning system 10 according to thepresent embodiment, it is possible to create an inference model havingan inference accuracy that satisfies the target accuracy. The machinelearning method using the machine learning system 10 according to thepresent embodiment is understood as an inference model creation method.

System Configuration Example

Next, an example of a specific configuration of the machine learningsystem 10 will be described. FIG. 4 is a diagram schematicallyillustrating a system configuration example of the machine learningsystem 10 according to the embodiment of the present invention. First,an example of a medical institution network 50 will be described. Forsimplicity of illustration, FIG. 4 illustrates an example in which themedical institution network 50 having the same system configuration isprovided in each of a plurality of medical institutions. However, amedical institution network having a different system configuration foreach medical institution may be provided.

The medical institution network 50 is a computer network including acomputed tomography (CT) apparatus 52, a magnetic resonance imaging (MM)apparatus 54, a computed radiography (CR) apparatus 56, a picturearchiving and communication systems (PACS) server 58, a CAD server 60, aterminal 62, and an internal communication line 64.

The medical institution network 50 is not limited to the CT apparatus52, the MM apparatus 54, and the CR apparatus 56 illustrated in FIG. 4.Instead of some or all of the apparatuses or in addition to theapparatuses, the medical institution network 50 may include at least oneor a combination of a digital X-ray imaging apparatus, an angiographyX-ray diagnosis apparatus, an ultrasound diagnostic apparatus, apositron emission tomography (PET) apparatus, an endoscopic apparatus, amammography apparatus, and various inspection apparatuses (modalities)which are not illustrated. There may be various combinations of types ofinspection apparatuses connected to the medical institution network 50for each medical institution. Each of the CT apparatus 52, the MRIapparatus 54, and the CR apparatus 56 is an example of the “imagingapparatus” in the present disclosure.

The PACS server 58 is a computer that stores and manages various dataand comprises a large-capacity external storage apparatus and databasemanagement software. The PACS server 58 performs a communication withanother apparatus via the internal communication line 64, and transmitsand receives various data including image data. The PACS server 58receives various data including image data and the like generated byeach inspection apparatus such as the CT apparatus 52, the MRI apparatus54, and the CR apparatus 56 via the internal communication line 64, andstores and manages the data in a recording medium such as alarge-capacity external storage apparatus.

A storage format of the image data and a communication between theapparatuses via the internal communication line 64 are based on aprotocol such as digital imaging and communication in medicine (DICOM).The PACS server 58 may be a DICOM server that operates according to aDICOM specification. The data stored in the PACS server 58 can be usedas learning data. The learning data created based on the data stored inthe PACS server 58 may be stored in the CAD server 60. The PACS server58 is an example of a “data storage apparatus of a medical institution”according to the present disclosure. Further, the CAD server 60 mayfunction as the “data storage apparatus of a medical institution”according to the present disclosure.

The CAD server 60 corresponds to the client 20 described in FIG. 1. TheCAD server 60 has a communication function for a communication with theintegration server 30 and is connected to the integration server 30 viaa wide area communication line 70. The CAD server 60 can acquire datafrom the PACS server 58 or the like via the internal communication line64. The CAD server 60 includes a local learning management program forexecuting training of the local model LM on the CAD server 60 using thedata group stored in the PACS server 58. The CAD server 60 is an exampleof a “client terminal” according to the present disclosure.

Various data stored in the database of the PACS server 58 and variousinformation including the inference result by the CAD server 60 can bedisplayed on the terminal 62 connected to the internal communicationline 64.

The terminal 62 may be a display terminal called a PACS viewer or aDICOM viewer. A plurality of terminals 62 may be connected to themedical institution network 50. A form of the terminal 62 is notparticularly limited and may be a personal computer, a workstation, atablet terminal, or the like.

As illustrated in FIG. 4, the medical institution network having thesame system configuration is provided in each of the plurality ofmedical institutions. The integration server 30 performs a communicationwith a plurality of CAD servers 60 via the wide area communication line70. The wide area communication line 70 is an example of a“communication line” according to the present disclosure.

Configuration Example of Integration Server 30

FIG. 5 is a block diagram illustrating a configuration example of theintegration server 30. The integration server 30 can be formed by acomputer system configured by using one or a plurality of computers. Theintegration server 30 is formed by installing a program on a computer.

The integration server 30 comprises a processor 302, a non-transitorytangible computer-readable medium 304, a communication interface 306, aninput/output interface 308, a bus 310, an input device 314, and adisplay device 316. The processor 302 is an example of a “firstprocessor” according to the present disclosure. The computer-readablemedium 304 is an example of a “first computer-readable medium” accordingto the present disclosure.

The processor 302 includes a central processing unit (CPU). Theprocessor 302 may include a graphics processing unit (GPU). Theprocessor 302 is connected to the computer-readable medium 304, thecommunication interface 306, and the input/output interface 308 via thebus 310. The input device 314 and the display device 316 are connectedto the bus 310 via the input/output interface 308.

The computer-readable medium 304 includes a memory as a main storageunit and a storage as an auxiliary storage device. The computer-readablemedium 304 may be, for example, a semiconductor memory, a hard diskdrive (HDD) device, a solid state drive (SSD) device, or a combinationof these devices.

The integration server 30 is connected to the wide area communicationline 70 (refer to FIG. 4) via the communication interface 306.

The computer-readable medium 304 includes a master model storage unit320, a verification data storage unit 322, an imaging apparatusinformation storage unit 324, and a database 326. The master modelstorage unit 320 stores data of the latest version of a master model MM.The verification data storage unit 322 stores a plurality of pieces ofverification data TD which are used when verifying the inferenceaccuracy of the integration model created by the master model candidatecreation unit 334. The verification data TD is data in which input dataand correct answer data are combined, and is also called test data. Theverification data TD may be, for example, data provided by a universityand the like. The verification data storage unit 322 stores theverification data TD prepared for each imaging condition. Therepresentations “imaging condition 1” and “imaging condition 2” in FIG.5 represent the imaging condition category, and the number at the endcorresponds to the index

The imaging apparatus information storage unit 324 stores imagingapparatus information of the learning client group. The imagingapparatus information includes a manufacturer name and model number ofthe apparatus which is the imaging apparatus.

The computer-readable medium 304 stores various programs including asynchronization program 328 and the master model learning managementprogram 33, and data. The synchronization program 328 is a program forproviding the data of the master model MM to each client 20 via thecommunication interface 306 and synchronizing each local model LM withthe master model MM. In a case where the processor 302 executes aninstruction of the synchronization program 328, the computer functionsas a synchronization processing unit. The synchronization program 328may be incorporated as the program module of the master model learningmanagement program 33.

In a case where the processor 302 executes an instruction of the mastermodel learning management program 33, the computer functions as a mastermodel learning management unit 330. The master model learning managementunit 330 includes the imaging condition classification unit 332, aclient cluster creation unit 333, the master model candidate creationunit 334, and an inference accuracy evaluation unit 340. The inferenceaccuracy evaluation unit 340 includes an inference unit 342, aninference accuracy calculation unit 344, and an accuracy target valuecomparison unit 346.

The client cluster creation unit 333 creates the client cluster which isthe combination of the clients 20 used to create the master modelcandidate MMC for each imaging condition category classified by theimaging condition classification unit 332. In a case where the clientcluster creation unit 333 creates the plurality of client clusters,there is no need to distribute all the clients of the client populationto one of the client clusters, and the learning results of some clients20 may not be used in the integration processing.

The creation of the client cluster by the client cluster creation unit333 may be performed before each client 20 starts learning, or may beperformed after learning is started. For example, the creation of theclient cluster may be performed after each learning result is receivedfrom each client 20. The communication interface 306 is an example of a“reception unit” according to the present disclosure.

The client cluster creation unit 333 stores, in the database 326,information indicating a correspondence relationship between theinformation of the clients 20 belonging to each client cluster and themaster model candidate MMC created for each client cluster. The database326 is an example of an “information storage unit” according to thepresent disclosure.

The master model candidate creation unit 334 creates a master modelcandidate MMC by integrating the learning results for each clientcluster. Information indicating a correspondence relationship as towhich client cluster each master model candidate MMC created is based onis stored in the database 326.

The inference accuracy evaluation unit 340 verifies and evaluates theinference accuracy of the master model candidate MMC created for eachclient cluster.

The inference unit 342 executes an inference by the master modelcandidate MMC by inputting the verification data TD to the master modelcandidate MMC. The inference accuracy calculation unit 344 calculates aninference accuracy of the master model candidate MMC by comparing theinference result of the master model candidate MMC obtained from theinference unit 342 with the correct answer data. For example, as thecorrect answer data, data in which the number of lesions and correctclinical findings are added to the image data is used. The inferenceaccuracy calculation unit 344 performs an accuracy verification aplurality of times through comparison with the verification data TD. Theinference accuracy calculation unit 344 may calculate an accuracyaverage value of the master model candidate MMC from the result obtainedby performing the accuracy verification a plurality of times, andevaluate the accuracy average value as the inference accuracy of themaster model candidate MMC. The inference accuracy calculated by theinference accuracy calculation unit 344 is stored in the database 326.

The accuracy target value comparison unit 346 selects the inferenceaccuracy of the model having the highest inference accuracy from theplurality of created master model candidates, and determines whether ornot the master model candidate having an inference accuracy higher thanthe accuracy target value is obtained by comparing the inferenceaccuracy with an accuracy target value. The accuracy target value is avalue indicating a target accuracy and is, for example, set to anaccuracy higher than the inference accuracy of the latest version of themaster model MM and is set to an accuracy having a level forcommercialization instead of the master model MM.

The synchronization program 328 and the master model learning managementprogram 33 are examples of a “first program” in the present disclosure.

Further, in a case where the processor 302 executes an instruction of adisplay control program, the computer functions as a display controlunit 350. The display control unit 350 generates a display signalrequired for a display output to the display device 316 and performs adisplay control of the display device 316.

The display device 316 is configured with, for example, a liquid crystaldisplay, an organic electro-luminescence (OEL) display, a projector, oran appropriate combination thereof. The input device 314 is configuredwith, for example, a keyboard, a mouse, a touch panel, another pointingdevice, a voice input device, or an appropriate combination thereof. Theinput device 314 receives various inputs from an operator. The displaydevice 316 and the input device 314 may be integrally configured byusing a touch panel.

The display device 316 can display the inference accuracy in eachlearning iteration of each of the plurality of master model candidatesMMC. That is, information indicating a learning progress status of eachof the plurality of master model candidate MMCs can be displayed on thedisplay device 316, and the information displayed on the display device316 enables the operator to confirm the learning progress status of eachmaster model candidate MMC.

Configuration Example of CAD Server 60

FIG. 6 is a block diagram illustrating a configuration example of theCAD server 60 as an example of the client 20. The CAD server 60 can beformed by a computer system configured by using one or a plurality ofcomputers. The CAD server 60 is formed by installing a program on acomputer.

The CAD server 60 comprises a processor 602, a non-transitory tangiblecomputer-readable medium 604, a communication interface 606, aninput/output interface 608, a bus 610, an input device 614, and adisplay device 616. The hardware configuration of the CAD server 60 maybe the same as the hardware configuration of the integration server 30described with reference to FIG. 5. That is, the hardware configurationof each of the processor 602, the computer-readable medium 604, thecommunication interface 606, the input/output interface 608, the bus610, the input device 614, and the display device 616 in FIG. 6 may bethe same as the hardware configuration of each of the processor 302, thecomputer-readable medium 304, the communication interface 306, theinput/output interface 308, the bus 310, the input device 314, and thedisplay device 316 in FIG. 5.

The CAD server 60 is an example of an “information processing apparatus”according to the present disclosure. The processor 602 is an example ofa “second processor” according to the present disclosure. Thecomputer-readable medium 604 is an example of a “secondcomputer-readable medium” according to the present disclosure.

The CAD server 60 is connected to a learning data storage unit 80 viathe communication interface 606 or the input/output interface 608. Thelearning data storage unit 80 is configured to include a storage thatstores learning data to be used for machine learning by the CAD server60. The “learning data” is training data used for machine learning andis synonymous with “data for the learning” or “training data”. Thelearning data stored in the learning data storage unit 80 is the localdata LD described with reference to FIG. 1. The learning data storageunit 80 may be the PACS server 58 described with reference to FIG. 2.The learning data storage unit 80 is an example of a “data storageapparatus of a medical institution” according to the present disclosure.

Here, an example in which the learning data storage unit 80 and the CADserver 60 that executes learning processing are configured as separateapparatuses will be described. However, the functions may be formed byone computer, or the processing functions may be shared and formed bytwo or more computers. The computer-readable medium 604 of the CADserver 60 illustrated in FIG. 6 stores various programs, which include alocal learning management program 630 and a diagnosis support program640, and data.

In a case where the processor 602 executes an instruction of the locallearning management program 630, the computer functions as asynchronization processing unit 631, a learning data classificationprocessing unit 632, a learning data acquisition unit 633, a local modelLM, an error calculation unit 634, an optimizer 635, a learning resultstorage unit 636, and a transmission processing unit 637. The locallearning management program 630 is an example of a “second program”according to the present disclosure.

The synchronization processing unit 631 performs a communication withthe integration server 30 via the communication interface 606 andsynchronizes the master model MM in the integration server 30 with thelocal model LM in the CAD server 60.

The learning data classification processing unit 632 classifies thelocal data LD stored in the learning data storage unit 80 into imagingcondition categories and divides the imaging condition categories into aset of learning data for each imaging condition category.

The learning data acquisition unit 633 acquires learning data from thelearning data storage unit 80. The learning data acquisition unit 633may be configured to include a data input terminal for receiving datafrom an external apparatus or from another signal processing unit in theapparatus. Further, the learning data acquisition unit 633 may beconfigured to include the communication interface 606, the input/outputinterface 608, a media interface for performing reading and writing on aportable external storage medium such as a memory card (notillustrated), or an appropriate combination of these interfaces.

The learning data acquired via the learning data acquisition unit 633 isinput to the local model LM as a learning model.

The error calculation unit 634 calculates an error between a predictedvalue indicated by a score which is output from the local model LM andthe correct answer data. The error calculation unit 634 evaluates theerror using a loss function. The loss function may be, for example, across entropy or a mean square error.

The optimizer 635 performs processing of updating a weight parameter ofthe local model LM from the calculation result of the error calculationunit 634. The optimizer 635 performs calculation processing of obtainingan update amount of the weight parameter of the local model LM andupdate processing of the weight parameter of the local model LMaccording to the calculated update amount of the weight parameter, byusing the error calculation result obtained from the error calculationunit 634. The optimizer 635 updates the weight parameter based on analgorithm such as a backpropagation.

The learning processing unit 638 including the learning data acquisitionunit 633, the local model LM, the error calculation unit 634, and theoptimizer 635 may be provided for each imaging condition categoryclassified by the learning data classification processing unit 632.

The CAD server 60 in which the local learning management program 630 isincorporated functions as a local learning apparatus that executesmachine learning on the CAD server 60 by using the local data LD aslearning data. The CAD server 60 reads the learning data, which is thelocal data LD, from the learning data storage unit 80 and executesmachine learning of each local model LM using the learning dataclassified into each imaging condition category. The CAD server 60 canupdate, in a case where the learning data is read in units of mini-batchin which a plurality of pieces of learning data are collected, theweight parameter.

The local learning management program 630 repeats an iteration of thelearning processing until a learning end condition is satisfied for eachlocal model LM. After the learning end condition is satisfied, theweight parameter of the local model LM is stored in the learning resultstorage unit 636 as the learning result.

The transmission processing unit 637 performs processing of transmittingthe learning result to the integration server 30. The weight parameterof the trained local model LM stored in the learning result storage unit636 is transmitted to the integration server 30 via the communicationinterface 606 and the wide area communication line 70 (refer to FIG. 4).The transmission processing unit 637 and the communication interface 606are examples of a “transmission unit” according to the presentdisclosure.

Further, in a case where the processor 602 executes an instruction ofthe diagnosis support program 640, the computer functions as an AI-CADunit 642.

The AI-CAD unit 642 outputs an inference result for input data by using,as an inference model, the master model MINI or the local model LM. Theinput data to the AI-CAD unit 642 is, for example, a medical image suchas a two-dimensional image, a three-dimensional image, and a movingimage, and an output from the AI-CAD unit 642 is, for example,information indicating a position of a lesion portion in the image,information indicating a class classification such as a disease name, ora combination thereof.

Description of Local Learning Management Program 630

As described above, the local learning management program 630 isinstalled on the client terminal (client 20) existing in the medicalinstitution network 50. Here, the client terminal may be, for example,the CAD server 60 in FIG. 4. The local learning management program 630has a function of synchronizing the master model MINI before performinglearning and the local model LM, a function of starting the locallearning, a function of setting an end condition of local learning, anda function of transmitting the result of local learning to theintegration server 30 when local learning is ended.

FIG. 7 is a flowchart illustrating an example of an operation of theclient terminal based on the local learning management program 630.Steps in the flowchart illustrated in FIG. 7 are executed by theprocessor 602 according to an instruction of the local learningmanagement program 630.

In step S21, the processor 602 of the CAD server 60 determines whetheror not the local model LM created for each imaging condition categoryexists on the CAD server 60. In a case where a determination result instep S21 is No, the processor 602 proceeds to step S22, collects andclassifies the required number of images for the learning for eachsimilar imaging condition from the local data LD, and creates thelearning data set for each imaging condition category. In step S23, theprocessor 602 creates the local model LM for each of the classifiedsimilar imaging conditions. After step S23, the processor 602 proceedsto step S24.

On the other hand, in a case where the determination result in step S21is Yes, that is, in a case where the local model LM already exists,steps S22 and S23 have already been executed, and the imageclassification for each similar imaging condition is performed.Therefore, the processor 602 proceeds to step S24.

In step S24, the processor 602 determines whether to synchronize thelocal model LM on the CAD server 60 with the latest version of themaster model MM on the integration server 30. The CAD server 60communicates with the integration server 30 before the learning isstarted to determine the necessity of the synchronization.

In a case where a determination result in step S24 is Yes, the processor602 proceeds to step S25 to synchronize each local model LM with themaster model MM.

For example, in a case where a new local model LM is created in stepS23, the local model LM is synchronized with the master model MM. In acase where the latest version of the master model MM used for thelearning exists on the integration server 30, the local model LM issynchronized with the latest version of the master model MM.

In this case, the relationship between the synchronized local model LMand the master model MM, which are synchronized, is maintained in thesubsequent learning process. The synchronization method may be a methodof downloading and updating a model parameter file of the master modelMM on a client side or may be a method of managing a virtual containerimage or the like of the master model MM on the integration server 30and downloading the virtual container image or the like to a terminalside, which is the client 20. With the synchronization processing, themaster model MINI is a learning model (local model LM) in an initialstate before the learning is started.

The processor 602 may synchronize the local model LM with the mastermodel MM, for example, at a point in time set by the local learningmanagement program 630. Here, a “set time” may be designated as a fixedvalue, for example, a time outside of hospital examination businesshours, or may be programmatically set by storing a record of anoperating status of the CAD server 60 and determining a time when theCAD server 60 is not normally used.

In a case where the determination is No after step S25 or in step S24,the processor 602 proceeds to step S26.

In step S26, the processor 602 executes the local learning for eachsimilar imaging condition. In the local model LM created for eachsimilar imaging condition, the learning processing of the local model LMis started by the local learning management program 630, and the locallearning proceeds using the learning data group for each imagingcondition category classified and collected from the local data LD inthe medical institution network 50.

In step S27, the processor 602 determines whether or not the learningend condition is satisfied. Here, examples of the learning end conditioninclude the following conditions.

[Example 1] The number of iterations is designated in advance, andlearning is ended after the designated number of iterations.

[Example 2] With the upper limit number of iterations as the upperlimit, the learning proceeds until the designated accuracy improvementis satisfied. That is, in a state where the verification data TD isstored in the medical institution network 50, the inference accuracy iscalculated by performing accuracy comparison between the inferenceresult obtained by inputting the verification data TD into the trainedmodel and the correct answer, and learning is performed until theaccuracy improvement of a designated ratio is achieved with the upperlimit number of iterations as the upper limit. In a case where theaccuracy improvement of the designated ratio is achieved within theupper limit number of iterations set in advance, the learning is ended.

[Example 3] A time limit is set, and the learning is performed withinthe time limit. In a case where the time limit is reached, the learningis ended.

The end condition of any one of [Example 1] to [Example 3] may be set,or a logical product (AND) or a logical sum (OR) of the plurality ofconditions may be set as the end condition.

In a case where a determination result in step S27 is No, the processor602 returns to step S26 to continue the local learning processing. Onthe other hand, in a case where the determination result in step S27 isYes, the processor 602 proceeds to step S28 to end the learning.

After the learning is ended, the processor 602 transmits the learningresult of the local model LM and the imaging condition information usedfor the learning to the integration server 30 in step S29. For example,the processor 602 stores the weight parameter of the trained model in afile and transmits the weight parameter thereof together with themetadata including the imaging condition information to the integrationserver 30 via the wide area communication line 70.

Each of the plurality of the CAD servers 60 illustrated in FIG. 4executes machine learning of each local model LM by using, as learningdata, data stored in the PACS server 58 in different medical institutionnetworks, and transmits a learning result to the integration server 30via the wide area communication line 70.

Description of Master Model Learning Management Program 33

FIG. 8 is a flowchart illustrating an example of an operation of theintegration server 30 based on the master model learning managementprogram 33. Steps in the flowchart illustrated in FIG. 8 are executed bythe processor 302 of the integration server 30 according to aninstruction of the master model learning management program 33.

In step S31, the processor 302 receives the learning result and themetadata accompanying the learning result from each client 20.

In step S32, the processor 302 associates and stores the learning resulttransmitted from each client 20 with the metadata.

In step S33, the processor 302 creates the classification of similarimaging conditions based on the stored learning result and metadata. Instep S34, the processor 302 creates the client cluster for each of theclassified imaging condition categories.

In step S35, the processor 302 integrates the learning results for eachclient cluster to create the master model candidate MMC from each clientcluster. In a case where the plurality of master model candidates MMCare created in step S35, the processor 302 stores the informationindicating the correspondence relationship as to from which clientcluster each master model candidate MMC is created in the data storageunit such as the database 326. The relationship between the clientcluster and the master model candidate MMC is maintained until thesubsequent training of the master model candidate MMC is ended.

In step S36, the processor 302 evaluates the inference accuracy for eachof the created master model candidates MMC. That is, the processor 302causes the master model candidate MMC to perform the inference by using,as an input, the verification data TD for each imaging conditionprepared in advance to calculate the inference accuracy, and comparesthe inference accuracy with the accuracy target value. Further, theprocessor 302 stores, in the database 326, the calculated inferenceaccuracy and the comparison result between the inference accuracy andthe accuracy target value in association (correlation) with the mastermodel candidate MMC.

For the inference accuracy of the master model candidate that is to becompared with the accuracy target value when performing processing ofstep S36, an instantaneous value or a statistical value such as anaverage value or a median value is used as an appropriate value. Anexample of processing contents in the inference accuracy evaluationapplied to step S36 will be described later with reference to FIG. 9.

In step S37, the processor 302 determines whether or not a master modelcandidate MMC having an inference accuracy higher than the accuracytarget value is obtained. In a case where a determination result in stepS37 is Yes, that is, in a case where the inference accuracy of themaster model candidate MMC exceeds the accuracy target value, theprocessor 302 ends the training of the master model candidate MMC (stepS38) and sends notification of the end of the learning thereof (stepS39).

For example, in step S38, the processor 302 sets a master modelcandidate MMC having the inference accuracy higher than the accuracytarget value, as the latest model having improved performance afterlearning, stores the model in the data storage unit such as the database326 in an appropriate format such as a file, and sends notification thatlearning is ended. Here, as a notification method, a message queue, ageneral inter-process communication, or the like may be used. Thenotification that the learning is ended may be displayed on the displaydevice 316 or may be transmitted to the client 20.

On the other hand, in a case where the determination result in step S37is No, that is, in a case where the master model candidate MMC having aninference accuracy higher than the accuracy target value is notobtained, the processor 302 proceeds to step S40.

In step S40, the processor 302 synchronizes, using the correspondenceinformation stored in step S35, the local model LM of the client clusterused to create the master model candidate MMC that has not reached thetarget accuracy with the master model candidate MMC, causes thecorresponding client 20 to repeat the local learning (refer to FIG. 7),and repeats steps S31 to S40.

In step S40, the processor 302 may set the master model candidate MMChaving the highest inference accuracy that is found in the repetition ofstep S31 to step S40 as a provisional master model and synchronize themodel with the local model LM of the client.

Example of Inference Accuracy Evaluation Processing

FIG. 9 is a flowchart illustrating an example of processing ofevaluating an inference accuracy of the master model candidate MMC inthe integration server 30. The flowchart illustrated in FIG. 9 isapplied to step S36 of FIG. 8. Here, the inference accuracy evaluationprocessing is described for one master model candidate MMC. However, thesame processing is performed for each master model candidate MMC createdfrom each of the plurality of client clusters in which the combinationof the clients 20 is different for each imaging condition category.

In step S341 of FIG. 9, the processor 302 causes the master modelcandidate MMC to execute the inference with the verification data TD foreach imaging condition as an input.

In step S342, the processor 302 calculates an inference accuracy of themaster model candidate MMC based on the inference result and the correctanswer data.

In step S343, the processor 302 compares the inference accuracy of themaster model candidate MMC with an accuracy target value. Here, theaccuracy target value may be compared with an instantaneous value of theinference accuracy of the master model candidate MMC. However, in thecomparison, while maintaining the configuration of the client clusterused for the creation of the master model candidate MMC, the proceduresof steps S31 to S343 may be performed for several iterations, theinference accuracy at that time may be recorded each time, and astatistical value such as an average value or a median value of theinference accuracy may be compared with the accuracy target value.

In step S344, the processor 302 stores the inference accuracy of themaster model candidate MMC and the comparison result between theinference accuracy and the accuracy target value in the database 326.

After step S344, the processor 302 ends the flowchart of FIG. 9 andreturns to the flowchart of FIG. 8.

Specific Example of Processing by Cooperation of Integration Server 30and Plurality of Clients 20

Here, a specific example of processing performed by the integrationserver 30 and the plurality of clients 20 will be described. In theexample, it is assumed that the plurality of clients 20 are a pluralityof CAD servers 60 illustrated in FIG. 4. The integration server 30 andthe plurality of CAD servers 60 execute processing of [Procedure 301] to[Procedure 307] to be described below.

[Procedure 301] The distribution learning client program is executed onthe CAD server 60 in the medical institution network 50 of each of theplurality of medical institutions.

[Procedure 302] The integration server 30 stores the imaging apparatusinformation of the learning client group. The integration server 30extracts the required number of a part of the client group (clientcluster) used for the learning for each similar imaging apparatus fromthe client group including the innumerable clients 20, which is theclient population, to create a plurality of client groups for eachsimilar imaging apparatus. Here, the term “for each similar imagingapparatus” may be, for example, an imaging apparatus group having thesame model number series of the apparatus. The classification “for eachsimilar imaging apparatus” is a form of the classification “for eachsimilar imaging condition”. Here, an example of classifying according tothe type (model) of the apparatus which is the “imaging apparatus” isdescribed. Similar imaging conditions may be defined according toconditions of a combination of the model and the imaging condition toclassify the imaging condition categories.

[Procedure 303] The client 20 for the distribution learning in eachclient cluster performs iterations for learning a set number of timesusing data (for example, a medical image) in the medical institutionnetwork to which the client 20 belongs and information accompanying thedata.

[Procedure 304] Each client 20 transmits the weight parameter of thetrained learning model to the integration server 30 via the wide areacommunication line 70.

An attached document of a medical apparatus as the client 20 that usesthe function according to the present embodiment describes that learningis performed as background processing within a range in which thelearning does not interfere with medical work. In addition, the attacheddocument describes that learning data to be used is data in the medicalinstitution, that data to be transmitted to the outside is only a weightparameter after learning, and that data by which an individual isidentified is not transmitted.

[Procedure 305] The integration server 30 collects the weight parametersof the learning results transmitted from the client 20 for each clientcluster, and creates a master model candidate MMC for each clientcluster.

[Procedure 306] The integration server 30 performs the accuracyverification of each master model candidate MMC created for each clientcluster. The accuracy verification may be paraphrased as “accuracyevaluation”. The integration server 30 causes the master model candidateMMC to perform an inference using the verification data TD and comparesan inference result with the correct answer data. The verification dataTD is prepared for each similar imaging apparatus. The accuracyverification is performed by using an image captured by an imagingapparatus that is the same as or similar to the imaging apparatus thatcaptures the image of the learning data used to create the master modelcandidate MMC.

[Procedure 307] The integration server 30 confirms the inferenceaccuracy of the model having the highest inference accuracy among themaster model candidates MMC created for each client cluster. In a casewhere the highest inference accuracy is higher than the target accuracy(accuracy target value), the master model candidate MMC having thehighest accuracy (maximum inference accuracy) is adopted as a productmodel.

On the other hand, in a case where the inference accuracy of the mastermodel candidate MMC with the highest accuracy is lower than the accuracytarget value, the integration server 30 performs the learning iterationfrom procedure 303 to procedure 307 again using the client cluster withhigher accuracy.

The integration server 30 performs the learning iteration from procedure303 to procedure 307 until the master model candidate MMC having theinference accuracy higher than the accuracy target value is obtained.Alternatively, in a case where the master model candidate MMC having aninference accuracy higher than the accuracy target value is not obtainedeven though iterations are performed by the designated upper limitnumber of iterations, the integration server 30 may adopt the mastermodel candidate MMC from which the maximum inference accuracy isobtained in the search process so far, as the product model.

In this way, in the new master model created by performing the machinelearning method using the machine learning system 10 according to thepresent embodiment, the inference accuracy is improved as compared withthe master model before the learning.

According to the present embodiment, it is possible to update aninference performance of the master model MM. In a case where the newmaster model created by performing the machine learning method accordingto the present embodiment is provided by sales or the like, preferably,the number of the clients used for the learning, the number of pieces ofverification data used for verification of the accuracy, and the likeare described in an attached document provided in sales. For the numberof the clients used for the learning, for example, the classification ofthe clients is preferably displayed such as “hospital_how many cases”,“clinic with bed_how many cases”, and “clinic without bed_how manycases” as a client profile.

As a preliminary procedure in a case where a version of the master modelas a current product is upgraded, information indicating the inferenceaccuracy in the previous version and the inference accuracy in the newversion and information indicating the number of the clients used foradditional learning and the classification of the clients are presentedto a medical institution, and an approval is received from the medicalinstitution before the version is upgraded. After the approval isobtained, the version is upgraded.

Example of Hardware Configuration of Computer

FIG. 10 is a block diagram illustrating an example of a hardwareconfiguration of a computer. A computer 800 may be a personal computer,a workstation, or a server computer. The computer 800 may be used as apart or all of the client 20, the integration server 30, the PACS server58, the CAD server 60, and the terminal 62 described above, or may beused as an apparatus having a plurality of functions thereof.

The computer 800 comprises a central processing unit (CPU) 802, a randomaccess memory (RAM) 804, a read only memory (ROM) 806, a graphicsprocessing unit (GPU) 808, a storage 810, a communication unit 812, aninput device 814, a display device 816, and a bus 818. The GPU 808 maybe provided as necessary.

The CPU 802 reads out various programs stored in the ROM 806, thestorage 810, or the like, and executes various processing. The RAM 804is used as a work area of the CPU 802. Further, the RAM 804 is used as astorage unit for temporarily storing the read program and various data.

The storage 810 includes, for example, a hard disk apparatus, an opticaldisk, a magneto-optical disk, a semiconductor memory, or a storagedevice configured by using an appropriate combination thereof. Thestorage 810 stores various programs, data, and the like required forinference processing and/or learning processing. The program stored inthe storage 810 is loaded into the RAM 804, and the CPU 802 executes theprogram. Thus, the computer 800 functions as means for performingvarious processing defined by the program.

The communication unit 812 is an interface that performs communicationprocessing with an external apparatus in a wired manner or a wirelessmanner and exchanges information with the external apparatus. Thecommunication unit 812 may play a role of an information acquisitionunit that receives an input such as an image.

The input device 814 is an input interface that receives variousoperation inputs to the computer 800. The input device 814 is configuredwith, for example, a keyboard, a mouse, a touch panel, another pointingdevice, a voice input device, or an appropriate combination thereof.

The display device 816 is an output interface for displaying variousinformation. The display device 816 is configured with, for example, aliquid crystal display, an organic electro-luminescence (OEL) display, aprojector, or an appropriate combination thereof.

Program for Operating Computer

A program causing a computer to realize a part or all of at least oneprocessing function among various processing functions, such as thelearning data classification function and the local learning function ineach client 20 and the master model learning management functionincluding the client cluster creation function, the master modelcandidate creation function, and the inference accuracy evaluationfunction in the integration server 30, described in the embodiment maybe recorded on a computer-readable medium as a non-transitory tangibleinformation storage medium, such as an optical disk, a magnetic disk, ora semiconductor memory, and the program may be provided with theinformation storage medium.

Further, instead of the form in which the program is provided by beingstored in a non-transitory tangible computer-readable medium, a programsignal may be provided as a download service using a telecommunicationline such as the Internet.

A service may be possible in which a part or all of at least oneprocessing function among the various processing functions, such as thelearning data classification function and the local learning functionand the master model learning management function including the clientcluster creation function, the master model candidate creation function,and the inference accuracy evaluation function, described in theembodiment is provided as an application server and the processingfunction is provided via a telecommunication line.

Hardware Configuration of Each Processing Unit

As a hardware structure of the processing units that execute variouspieces of processing, such as the imaging condition classification unit332 and the master model candidate creation unit 334 illustrated in FIG.3, the master model storage unit 320, the verification data storage unit322, the imaging apparatus information storage unit 324, the mastermodel learning management unit 330, the client cluster creation unit333, the inference accuracy evaluation unit 340, the inference unit 342,the inference accuracy calculation unit 344, the accuracy target valuecomparison unit 346, and the display control unit 350 illustrated inFIG. 5, and the synchronization processing unit 631, the learning dataclassification processing unit 632, the learning data acquisition unit633, the local model LM, the error calculation unit 634, the optimizer635, the learning result storage unit 636, the transmission processingunit 637, the AI-CAD unit 642, and the display control unit 650illustrated in FIG. 6, for example, the following various processors maybe used.

The various processors include a CPU which is a general-purposeprocessor that functions as various processing units by executing aprogram, a GPU which is a processor specialized for image processing, aprogrammable logic device (PLD) such as a field programmable gate array(FPGA) which is a processor capable of changing a circuit configurationafter manufacture, a dedicated electric circuit such as an applicationspecific integrated circuit (ASIC) which is a processor having a circuitconfiguration specifically designed to execute specific processing, andthe like.

One processing unit may be configured by one of these various processorsor may be configured by two or more processors having the same type ordifferent types. For example, one processing unit may be configured by aplurality of FPGAs, a combination of a CPU and an FPGA, or a combinationof a CPU and a GPU. Further, the plurality of processing units may beconfigured by one processor. As an example in which the plurality ofprocessing units are configured by one processor, firstly, asrepresented by a computer such as a client and a server, a form in whichone processor is configured by a combination of one or more CPUs andsoftware and the processor functions as the plurality of processingunits may be adopted. Secondly, as represented by a system on chip (SoC)or the like, a form in which a processor that realizes the function ofthe entire system including the plurality of processing units by oneintegrated circuit (IC) chip is used may be adopted. As described above,the various processing units are configured by using one or more variousprocessors as a hardware structure.

Further, as the hardware structure of the various processors, morespecifically, an electric circuit (circuitry) in which circuit elementssuch as semiconductor elements are combined may be used.

Advantages According to Present Embodiment

According to the machine learning system 10 according to the embodimentof the present invention, the following advantages are obtained.

[1] Learning can be performed without extracting personal informationsuch as a diagnosis image that requires consideration for privacy from amedical institution.

[2] A data group used for learning is classified into a data group withsubstantially uniform conditions (imaging condition category grouped assimilar imaging conditions) from the viewpoint of imaging conditions,and the learning is performed using homogenized learning data.Therefore, effective learning is possible.

[3] In federated learning, there is provided a mechanism for optimizingthe combination of the clients 20 used for creation of a new model byintegrating the learning results. Accordingly, as compared with a methodof collecting and integrating the learning results of all the clients 20into one or a method of randomly extracting a combination from theclient population, it is possible to select a client group that iseffective in improving the accuracy and to realize the high inferenceaccuracy at an early stage.

[4] It is possible to create an AI model having a high inferenceaccuracy.

MODIFICATION EXAMPLE 1

In the embodiment, the AI model for medical image diagnosis has beendescribed as an example. However, the scope of application of thetechnique of the present disclosure is not limited to this example. Forexample, the present disclosure may be applied even in a case wherelearning is performed on an AI model using time-series data as inputdata or an AI model using document data as input data. The time-seriesdata may be, for example, electrocardiogram waveform data. The documentdata may be, for example, a diagnosis report, and the present disclosuremay be applied to training of an AI model for supporting creation of areport. The electrocardiogram waveform data is an example of “inspectiondata” in the present disclosure.

The data used for the learning may be a combination of different typesof data acquired by different modalities. The condition category set as“the same or a similar condition” in the case may be set depending onthe combination of data acquisition conditions in the differentmodalities.

The data used for the learning may be a combination of a plurality oftypes of different data, such as a combination of images and time-seriesdata or a combination of images and document data. In a case where dataother than the image, such as electrocardiogram waveform data, is usedfor the learning, the terms “similar imaging condition”, “imagingcondition category”, and “imaging apparatus” described in the embodimentcan be extended to terms such as “similar inspection condition”,“inspection condition category”, and “inspection apparatus”, and theextended terms can be applied to the configuration of the embodiment.

MODIFICATION EXAMPLE 2

In the embodiment, an example in which an accuracy target value bylearning is set and the inference accuracy of the master model candidateis compared with the accuracy target value has been described. However,the accuracy target value may be updated as necessary. Instead of or incombination with the comparison with the accuracy target value, thelearning may proceed under a condition that the inference accuracy ofthe model is maximized within the time limit or the designated number ofiterations.

Other

The matters described in the configuration and the modification exampledescribed in the embodiment may be used in combination as appropriate,and some matters may be replaced. The present invention is not limitedto the embodiment described above, and various modifications may be madewithout departing from the scope of the present invention.

EXPLANATION OF REFERENCES

-   10: machine learning system-   20: client-   21: local learning management program-   30: integration server-   33: master model learning management program-   50: medical institution network-   52: CT apparatus-   54: MRI apparatus-   56: CR apparatus-   58: PACS server-   60: CAD server-   62: terminal-   64: internal communication line-   70: wide area communication line-   80: learning data storage unit-   302: processor-   304: computer-readable medium-   306: communication interface-   308: input/output interface-   310: bus-   314: input device-   316: display device-   320: master model storage unit-   322: verification data storage unit-   324: imaging apparatus information storage unit-   326: database-   328: synchronization program-   330: master model learning management unit-   332: imaging condition classification unit-   333: client cluster creation unit-   334: master model candidate creation unit-   340: inference accuracy evaluation unit-   342: inference unit-   344: inference accuracy calculation unit-   346: accuracy target value comparison unit-   350: display control unit-   602: processor-   604: computer-readable medium-   606: communication interface-   608: input/output interface-   610: bus-   614: input device-   616: display device-   630: local learning management program-   631: synchronization processing unit-   632: learning data classification processing unit-   633: learning data acquisition unit-   634: error calculation unit-   635: optimizer-   636: learning result storage unit-   637: transmission processing unit-   640: diagnosis support program-   642: AI-CAD unit-   650: display control unit-   800: computer-   802: CPU-   804: RAM-   806: ROM-   808: GPU-   810: storage-   812: communication unit-   814: input device-   816: display device-   818: bus-   CL1 to CL4, CLN, CLN+1: client-   LD, LD1, LD11, LD12: local data-   LD2, LD21, LD23, LD24: local data-   LD3, LD32, LD33: local data-   LM, LM11, LM12: local model-   LM21, LM23, LM24, LM32, LM33: local model-   MM: master model-   MMC: master model candidate-   TD: verification data-   S21 to S25: steps of local learning management processing-   S31 to S40: steps of master model learning management processing-   S341 to 5344: steps of inference accuracy evaluation processing

What is claimed is:
 1. A machine learning system comprising: a pluralityof client terminals; and an integration server, wherein each of theplurality of client terminals includes a terminal-side processorconfigured to: classify data stored in a data storage apparatus of amedical institution based on an acquisition condition of the data toclassify learning data into each data group acquired under the same or asimilar acquisition condition, the acquisition condition includingcondition concerning apparatus used to generate the data; executemachine learning of a learning model for each learning data groupclassified into each condition category of the same or a similaracquisition condition; and transmit learning results of the learningmodel executed for each learning data group and condition informationregarding the acquisition condition of the learning data group used forthe learning, to the integration server, and the integration serverincludes a trained master model, and a server-side processor configuredto: synchronize the learning model of each client terminal side with themaster model before each of the plurality of client terminals trains thelearning model; receive the learning results of the learning model andthe condition information from each of the plurality of clientterminals; classify the learning results into each condition category;integrate the learning results for each condition category to create aplurality of master model candidates; and evaluate an inference accuracyof each of the plurality of master model candidates.
 2. The machinelearning system according to claim 1, wherein the data includes an imagecaptured by using an imaging apparatus, and the acquisition conditionincludes an imaging condition for the image.
 3. The machine learningsystem according to claim 2, wherein the imaging condition includes acondition regarding a model of the imaging apparatus used for imaging.4. The machine learning system according to claim 2, wherein the imagingcondition includes a condition of an imaging parameter settable at atime of imaging.
 5. The machine learning system according to claim 1,wherein the data includes inspection data acquired by using aninspection apparatus, and the acquisition condition includes aninspection condition under which the inspection data is acquired.
 6. Themachine learning system according to claim 5, wherein the inspectioncondition includes condition concerning the inspection apparatus usedfor inspection.
 7. The machine learning system according to claim 5,wherein the inspection condition includes condition concerninginspection parameter settable at a time of inspection.
 8. The machinelearning system according to claim 1, wherein the acquisition conditionincludes a condition regarding a value of a parameter settable in anapparatus used to obtain the data, and the terminal-side processorclassifies acquisition conditions in which a specific value of theparameter, which is a specific acquisition condition, is within adesignated value range into the condition category in which theacquisition conditions are handled as the acquisition condition that isthe same as or similar to the specific acquisition condition.
 9. Themachine learning system according to claim 1, wherein a combination ofconditions in which a plurality of acquisition conditions are handled asa similar condition is designated, and the terminal-side processorperforms the classification into the condition category according to asetting of the designated similar condition.
 10. The machine learningsystem according to claim 1, wherein each of the plurality of clientterminals is a terminal provided in a medical institution network of adifferent medical institution.
 11. The machine learning system accordingto claim 1, wherein the integration server is provided in a medicalinstitution network or outside the medical institution network.
 12. Themachine learning system according to claim 1, wherein the learningresults transmitted from the client terminal to the integration serverinclude a weight parameter of the learning model after the learning. 13.The machine learning system according to claim 1, wherein the data usedas the learning data includes at least one type of data among atwo-dimensional image, a three-dimensional image, a moving image,time-series data, or document data.
 14. The machine learning systemaccording to claim 1, wherein each model of the learning model, themaster model, and the master model candidate is configured by using aneural network.
 15. The machine learning system according to claim 1,wherein the data used as the learning data includes a two-dimensionalimage, a three-dimensional image, or a moving image, and each model ofthe learning model, the master model, and the master model candidate isconfigured by using a convolutional neural network.
 16. The machinelearning system according to claim 1, wherein the data used as thelearning data includes time-series data or document data, and each modelof the learning model, the master model, and the master model candidateis configured by using a recurrent neural network.
 17. The machinelearning system according to claim 1, wherein the integration serverincludes an information storage that stores information indicating acorrespondence relationship as to which client cluster among a pluralityof client clusters each of the plurality of master model candidatescreated is based on.
 18. The machine learning system according to claim1, wherein the integration server includes a display on whichinformation indicating a progress status of learning of each of themaster model candidates is displayed.
 19. The machine learning systemaccording to claim 1, further comprising: a verification data storagethat stores verification data classified based on a data acquisitioncondition, wherein the server-side processor evaluates the inferenceaccuracy of the master model candidate using the verification data. 20.The machine learning system according to claim 1, wherein theserver-side processor is further configured to compare an inferenceresult output from the master model candidate by inputting verificationdata to the master model candidate with correct answer data of theverification data and calculate the inference accuracy of the mastermodel candidate, and compare the inference accuracy of the master modelcandidate with an accuracy target value.
 21. A machine learning methodusing a plurality of client terminals and an integration server, themethod comprising: classifying, via each of the plurality of clientterminals, data stored in a data storage apparatus of each of differentmedical institutions based on an acquisition condition of the data toclassify learning data into each data group acquired under the same or asimilar acquisition condition, the acquisition condition includingcondition concerning apparatus used to generate the data; synchronizinga learning model of each client terminal side with a trained mastermodel stored in the integration server before each of the plurality ofclient terminals trains the learning model; executing, via each of theplurality of client terminals, machine learning of the learning modelfor each learning data group classified into each condition category ofthe same or a similar acquisition condition; transmitting, via each ofthe plurality of client terminals, learning results of the learningmodel executed for each learning data group and condition informationregarding the acquisition condition of the learning data group used forthe learning, to the integration serve; via the integration server,receiving the learning results of the learning model and the conditioninformation from each of the plurality of client terminals; classifyingthe learning results into each condition category; integrating thelearning results for each condition category to create a plurality ofmaster model candidates; and evaluating an inference accuracy of each ofthe plurality of master model candidates.
 22. An integration serverconnected to a plurality of client terminals via a communication line,the integration server comprising: a first processor; and a firstcomputer-readable medium, which is a non-transitory tangible medium, onwhich a first program executed by the first processor is recorded,wherein the first processor executes, according to an instruction of thefirst program, processing including storing a trained master model onthe first computer-readable medium, synchronizing a learning model ofeach client terminal side with the master model before each of theplurality of client terminals trains the learning model, receivinglearning results of the learning model and condition informationregarding an acquisition condition of data included in a learning datagroup used for the learning from each of the plurality of clientterminals, the acquisition condition including condition concerningapparatus used to generate the data, classifying the learning resultsinto each condition category in which the acquisition condition ishandled as the same or a similar condition, integrating the learningresults for each condition category to create a plurality of mastermodel candidates, and evaluating an inference accuracy of each of theplurality of master model candidates.
 23. A non-transitory,computer-readable tangible recording medium that stores thereon acommand which causes, in a case where the command is read by a computer,the computer to realize the functions of: storing a trained mastermodel; synchronizing a learning model of each client terminal side withthe master model before each of the plurality of client terminals trainsthe learning model; receiving learning results of the learning model andcondition information regarding an acquisition condition of dataincluded in a learning data group used for the learning from each of theplurality of client terminals, the acquisition condition includingcondition concerning apparatus used to generate the data; classifyingthe learning results into each condition category in which theacquisition condition is handled as the same or a similar condition;integrating the learning results for each condition category to create aplurality of master model candidates; and evaluating an inferenceaccuracy of each of the plurality of master model candidates.
 24. Aninformation processing apparatus used as a client terminal connected toan integration server via a communication line, the informationprocessing apparatus comprising: a second processor; and a secondcomputer-readable medium, which is a non-transitory tangible medium, onwhich a second program executed by the second processor is recorded,wherein the second processor executes, according to an instruction ofthe second program, processing including classifying data stored in adata storage apparatus of a medical institution based on an acquisitioncondition of the data to classify learning data into each data groupacquired under the same or a similar acquisition condition, theacquisition condition including condition concerning apparatus used togenerate the data, executing, with a learning model synchronized with amaster model stored in the integration server as the learning model inan initial state before learning starts, machine learning of thelearning model for each learning data group classified into eachcondition category of the same or a similar acquisition condition, andtransmitting learning results of the learning model executed for eachlearning data group and condition information regarding the acquisitioncondition of the learning data group used for the learning, to theintegration server.
 25. A non-transitory, computer-readable tangiblerecording medium that stores thereon a command which causes, in a casewhere the command is read by a computer, the computer to realize thefunctions of: classifying data stored in a data storage apparatus of amedical institution based on an acquisition condition of the data toclassify learning data into each data group acquired under the same or asimilar acquisition condition, the acquisition condition includingcondition concerning apparatus used to generate the data; executing,with a learning model synchronized with a master model stored in theintegration server as the learning model in an initial state beforelearning starts, machine learning of the learning model for eachlearning data group classified into each condition category of the sameor a similar acquisition condition; and transmitting learning results ofthe learning model executed for each learning data group and conditioninformation regarding the acquisition condition of the learning datagroup used for the learning, to the integration server.
 26. A method ofcreating an inference model by performing machine learning methodaccording to claim 21, using a plurality of client terminals and anintegration server, the inference model creation method comprisingcreating an inference model with higher inference accuracy than themaster model based on a model whose inference accuracy satisfies atarget accuracy among the plurality of master model candidates.